Variational Learning : From exponential families to multilinear systems
نویسنده
چکیده
This note aims to give a general overview of variational inference on graphical models. Starting with the need for the variational approach, we proceed to the derivation of the Variational Bayes EM algorithm that creates distributions on the hidden variables in a graphical model. This leads us to the Variational message Passing algorithm for conjugate exponential families, which is shown to result in a set of updates for the parameters of the distributions involved. The updates form an iterative solution to a multilinear system involving the parameters of the exponential distributions.
منابع مشابه
Graphical Models, Exponential Families, and Variational Inference
The formalism of probabilistic graphical models provides a unifying framework for capturing complex dependencies among random variables, and building large-scale multivariate statistical models. Graphical models have become a focus of research in many statistical, computational and mathematical fields, including bioinformatics, communication theory, statistical physics, combinatorial optimizati...
متن کاملGeneralised Exponential Families and Associated Entropy Functions
A generalised notion of exponential families is introduced. It is based on the variational principle, borrowed from statistical physics. It is shown that inequivalent generalised entropy functions lead to distinct generalised exponential families. The well-known result that the inequality of Cramér and Rao becomes an equality in the case of an exponential family can be generalised. However, thi...
متن کاملVariational Bayesian Stochastic Complexity of Mixture Models
The Variational Bayesian framework has been widely used to approximate the Bayesian learning. In various applications, it has provided computational tractability and good generalization performance. In this paper, we discuss the Variational Bayesian learning of the mixture of exponential families and provide some additional theoretical support by deriving the asymptotic form of the stochastic c...
متن کاملInferring Block Structure of Graphical Models in Exponential Families
Learning the structure of a graphical model is a fundamental problem and it is used extensively to infer the relationship between random variables. In many real world applications, we usually have some prior knowledge about the underlying graph structure, such as degree distribution and block structure. In this paper, we propose a novel generative model for describing the block structure in gen...
متن کاملDeep Exponential Families
We describe deep exponential families (DEFs), a class of latent variable models that are inspired by the hidden structures used in deep neural networks. DEFs capture a hierarchy of dependencies between latent variables, and are easily generalized to many settings through exponential families. We perform inference using recent “black box” variational inference techniques. We then evaluate variou...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2005